Distance metrics

Claude Shannon’s Entropy

🔍 Information Theory Applications in Finance

• As it turns out, information theory in general, and the concept of Shannon’s entropy in particular, also have useful applications in finance. 💡

• The key idea behind entropy is to quantify the amount of uncertainty associated with a random variable. 🔍

• Information theory is also essential to ML, because the primary goal of many ML algorithms is to reduce the amount of uncertainty involved in the solution to a problem. 💯

👉 We will see how Shannon’s entropy, a key concept in information theory, can help solve real-world financial problems! 📈

Correlation is not a metric

Consider Two Random Vectors…

• Let X and Y be two random vectors of size T, and a correlation estimate ρ(X,Y), with the only requirement that σ(X,Y) = ρ(X,Y)σ(X)σ(Y).

• σ(X,Y) is the covariance between the two vectors.

• σ(X) and σ(Y) are the standard deviations of X and Y, respectively.

Pearson’s Correlation…

• Pearson’s correlation is one of several correlation estimates that satisfy these requirements.

An Alternative Correlation-Based Distance Metric

Entropy and forecasting complexity

• Similarly, if we add another kind of event to the distribution,forecasting into winter, so also predicting snow, entropy tends to increase, due the added dimensionality of the prediction problem.
• Similarly, if we add another kind of event to the distribution, forecasting into winter, so also predicting snow, entropy tends to increase, due the added dimensionality of the prediction problem.

p<-c(0.7,0.15,0.15)
-sum(p*log(p))

[1] 0.8188085

• Then entropy is about 0.82

Properties of Joint Entropy

• Non-negativity: \(H(X,Y) \ge 0\).
• Symmetry: \(H(X,Y) = H(Y,X)\).
• Greater than individual entropies: \(H(X,Y) \ge max(H(X),H(Y))\).
• Less than or equal to the sum of individual entropies: \(H(X,Y) \le H(X) + H(Y)\).

Calculating Joint Entropy

# Calculate mutual information (and hence joint entropy)
joint_entropy = mutinformation(discrete_returns1, discrete_returns2, method = "emp")
print(paste("Joint entropy (natural log base):", joint_entropy))

[1] "Joint entropy (natural log base): 0.194283857148803"

• If you specifically need the entropy value in bits, you can manually convert the output from natural logarithm to log base 2 by dividing the result by log(2):

# Convert the result to bits
joint_entropy_bits = joint_entropy / log(2)
print(paste("Joint entropy (in bits):", joint_entropy_bits))

[1] "Joint entropy (in bits): 0.280292357233359"

• Joint entropy provides insights into the information contained within the combined behavior of two financial assets.
• This example used simulated data and basic discretization for illustration.

Economic Interpretation of Joint Entropy in Finance

• Complexity of Asset Relationships: A joint entropy of \(\approx 0.28\) bits suggests a moderate level of unpredictability and complexity in the relationship between the two assets’ returns. This indicates some degree of informational efficiency but also room for diversification.
• Market Behaviour: The specific value reflects the combined information content and interdependency of the assets under study, implying that while they share some information, each still possesses unique characteristics.

Strategic Implications

• Diversification Benefits: The calculated joint entropy value points towards potential diversification benefits. It suggests that the assets’ price movements are not perfectly correlated, which can help in reducing overall portfolio volatility.
• Risk Management and Investment Decisions: This joint entropy value should inform risk management strategies by highlighting the importance of considering the complexity and predictability of asset returns. Investors should use this alongside other analyses for a comprehensive investment strategy.

Understanding Entropy in Finance

• Conditional Entropy:
  • Formula: (H(X|Y) = H(X,Y) - H(Y))
  • Represents the expected uncertainty in one variable ((X)) given the knowledge of another ((Y)). Essential for modeling predictability in financial markets.
• Maximum Entropy (MaxEnt):
  • Approach: Maximize (H(X)) subject to known constraints.
  • Identifies the least presumptive probability distributions based on known information, central to Bayesian inference and machine learning.

Understanding Entropy in Finance

Divergence and Its Financial Significance

• Kullback-Leibler (KL) Divergence:
  • Formula: (D_{KL}(p, q) = _i p_i ())
  • Measures the extra uncertainty induced when using distribution (q) to approximate (p), crucial for assessing model accuracy.
• Cross-Entropy:
  • Formula: (H_c(p, q) = -_{xS_X}p(x) (q(x)))
  • Highlights the discrepancy between the true distribution ((p)) and the predicted one ((q)), essential for evaluating classification models in finance. ## Understanding Entropy in Finance {.small} #### Mutual Information and Strategic Insights
• Mutual Information:
  • Formula: (I(X;Y) = H(X) - H(X|Y) = H(X) + H(Y) - H(X,Y))
  • Quantifies the informational gain in variable (X) from knowing (Y), demonstrating the value of shared information between financial variables for strategic decision-making.

Understanding Entropy in Finance

Implications for Financial Machine Learning

• Model Selection and Evaluation: Theoretical concepts like divergence and cross-entropy help in selecting and evaluating models based on their fidelity to true market behaviors and predictive capabilities.
• Strategic Decision-Making: Insights from mutual information and maximum entropy support financial practitioners in crafting robust investment strategies, managing risk, and optimizing portfolios with a principled understanding of market uncertainty.

Overview of Variation of Information (VI)

• Definition: A metric that quantifies the dissimilarity between two random variables or clusterings by combining mutual information (I) and entropy (H).
• Formula: (VI(X, Y) = H(X) + H(Y) - 2I(X; Y))
• Interpretation: Measures the total amount of uncertainty or information that is unique to each of the variables (X) and (Y), not shared between them.

Benefits and Challenges

TL;DR

Experimental evidence

• To calculate the Variation of Information (VI) in R and compare it with linear correlation, especially emphasizing the differences in detecting non-linear relationships, we’ll first simulate data to create an exaggerated non-linear relationship.
• Then, we’ll calculate both the Pearson correlation coefficient (a measure of linear correlation) and the Variation of Information.

Experimental evidence

First, let’s simulate the data:

set.seed(123) # Ensure reproducibility
x <- seq(-10, 10, by = 0.1)
y <- sin(x)^2 + rnorm(length(x), sd = 0.1) # Non-linear relationship with added noise
plot(x, y, main = "Simulated Non-linear Relationship", xlab = "X", ylab = "Y")

Calculating Pearson Correlation in R

The Pearson correlation can be directly calculated using the cor() function:

linear_correlation <- cor(x, y)
print(paste("Linear Correlation:", linear_correlation))

[1] "Linear Correlation: -0.00696311389621409"

Calculating Variation of Information in R

• Calculating the Variation of Information (VI) in R for continuous variables directly is not as straightforward because it typically requires the variables to be discretized or involves estimating their joint and individual entropies.

• Since there’s no built-in function for VI in base R or common packages, we’ll need to implement it. This involves discretizing the data, calculating the entropies, and then using the VI formula.

Calculating Variation of Information in R

library(entropy) # Load the infotheo package for entropy and mutual information calculations
x_disc <- discretize(x,numBins = 20) # Discretize x into 10 bins
y_disc <- discretize(y, numBins=20) # Discretize y into 10 bins

# Convert discretized data to numeric indices
x_bins <- as.numeric(factor(x_disc))
y_bins <- as.numeric(factor(y_disc))

# Create contingency table for joint distribution
joint_distribution <- table(x_bins, y_bins)

# Calculate individual entropies
H_x <- entropy(table(x_bins))
H_y <- entropy(table(y_bins))

# Calculate joint entropy
H_xy <- entropy(joint_distribution)

# Calculate mutual information
I_xy <- H_x + H_y - H_xy

# Calculate Variation of Information
VI <- H_x + H_y - 2 * I_xy
print(paste("Variation of Information:", VI))

[1] "Variation of Information: 2.23006570879938"

# For the Pearson correlation, no change is needed
linear_correlation <- cor(x, y)
print(paste("Linear Correlation:", linear_correlation))

[1] "Linear Correlation: -0.00696311389621409"

Normalized Variation of Information

• One common approach is to normalize VI by the joint entropy H(X,Y)H(X,Y) or by the sum of the individual entropies H(X)+H(Y)H(X)+H(Y), depending on the context and what aspect of the data you’re most interested in:

• Normalized VI=VI(X,Y)H(X)+H(Y)Normalized VI=H(X)+H(Y)VI(X,Y)​

• This normalization ensures that the VI lies between 0 and 1, where 0 indicates that the two variables share all their information (identical distributions) and 1 indicates that the two variables share no information.

Normalized Variation of Information in R

• Let’s continue from the previous example, assuming H_x, H_y, and I_xy (mutual information) have already been calculated:

# Assuming H_x, H_y, and I_xy have already been calculated

# Calculate Variation of Information (VI) again for clarity
VI <- H_x + H_y - 2 * I_xy

# Normalize VI to range between 0 and 1
# Normalized VI is VI divided by the sum of the individual entropies (max possible VI)
Normalized_VI <- VI / (H_x + H_y)

print(paste("Variation of Information:", VI))

[1] "Variation of Information: 2.23006570879938"

print(paste("Normalized Variation of Information:", Normalized_VI))

[1] "Normalized Variation of Information: 0.896160537565619"

::: ::: column #### Interpretation - Variation of Information (VI): Provides a measure of the total unique and shared information between two variables. A lower value indicates more shared information, while a higher value indicates less shared information and more uniqueness.

• Normalized Variation of Information: Adjusts VI into a 0 to 1 scale, where 0 indicates perfect agreement (identical information content) and 1 indicates complete disagreement (no shared information). This normalization makes it easier to interpret the degree of dissimilarity between the variables. ::: :::

::: :::

Practical Advice

• This normalisation technique does not change the essence of what VI measures but scales its output to a more interpretable range, especially useful when comparing across different pairs of variables or datasets.

• Remember, this normalised VI still does not indicate directionality or the type of relationship (linear or non-linear) between the variables, unlike correlation coefficients.

Explaining the Results

• This example simulates a non-linear relationship between (x) and (y), calculates the linear correlation to show its limitation in capturing non-linear dependencies, and calculates the Variation of Information, which is not limited to linear relationships and can capture the total amount of shared and unique information between the variables.

• The Pearson correlation coefficient might be low or not significant because it only measures linear relationships. In contrast, the Variation of Information could be higher, reflecting the complexity of the non-linear relationship that the linear correlation fails to capture.

Transfer entropy

• Financial time series dynamics are at the core of many interesting questions in quantitative finance
• In many cases this flow of information between markets can be approximate using these dynamics
• As useful extract of the previous concepts in this area is Transfer entropy which measures the amount of directed (time-asymmetric) transfer of information between two random processes.
• Transfer entropy is conditional mutual information with the history of the influenced variable as condition.
• In standard econometrics, transfer entropy is the non-parametric analogue to granger casuality from a vector autoregressive process.
• It generalises to allow for non-linear signal analysis -We will use the RTransferEntropy to investigate (Simon et al. 2019)

Information flows in financial markets

• Quantifying information flow between markets and individual stocks is a central tenant of quantitative research

• CAPM prices stocks in this way, imposing a linear unidirectional relationship between individual stocks and the market return

• Is this assumption valid?

• We will test this assumption using a selection of 10 stocks from the S&P 500

Information flow in financial markets

• Due to the need for Monte Carlo bootstrapping the estimation can be run over multicores.
• you will need a machine with at least 4 cores to run this in a reasonable time
• Using 64 cores the code below ran in 10 minutes.

library(RTransferEntropy)
library(future)
library(tidyverse)
# enable parallel processing
plan(multisession) # initialised a multicore enviroment
data("stocks") # loads data 
stocks %>% glimpse()
TE<-stocks %>% 
  group_by(ticker) %>% 
  group_split(.keep = TRUE) %>%
  map(~transfer_entropy(x = .x$ret,y=.x$sp500,shuffles = 500,type = "bins",bins=12)) 
names(TE)<-unique(stocks$ticker)
save(TE,file="Estimation_cache/Transfer_Entropy_SP.RData")

Information flows in financial markets

• The results are stored in a list of dataframes, one for each stock
• We can now visualise the results

• The results indicate that there is indeed a bi-directional flow of information present.
• Notably, there are some significant differences in the flows, for example AMD (Advance Micro Devices) contributes much more information to the market than it receives from it.

Conclusion

• Correlations are useful at quantifying the linear codependency between random variables.

• However,when variables X and Y are bound by a nonlinear relationship, the above distance metric misjudges the similarity of these variables.

• For non-linear cases, López de Prado (2020) argues the normalized mutual information is a more appropriate distance metric.

  • It allows us to answer questions regarding the unique information contributed by a random variable, without having to make functional assumptions.

• Given that many ML algorithms do not impose a functional form on the data, it makes sense to use them in conjunction with entropy-based features.